Skip to main content

Incident Report: Unwanted Wallet Balance Alert on Polygon Testnet for JOE-USD

Date: 2024-04-06
Time: 20:40 (UTC+05:30)
Duration: 1 day

Description​

Unwanted Wallet Balance Alert on Polygon Testnet even though the wallet balance is larger than required.

Root Cause​

There is a bug in the code. It is most probably because of comparing string values but should’ve turned them into a float numbers before comparing.

Impact​

False critical wallet balance alert. Need to be improved during mainnet launch.

Timeline​

  • 20:40 - First noticed the issue.
  • 20:45 - Initial diagnosis.
  • 17:49 (Day 2) - Started the fix.
  • 22:00 (Day 2) - Issue resolved.

Lessons Learned​

As a general note dumping the JSON in the alert description is not a good idea. Nobody noticed this because the JSON wasn’t readable enough. It should inform of the condition that created the alert and its extent. So if the condition is balance < expected balance, there should be a message that says "Balance lower than expected balance (by X)" (plus the dAPI name, address, etc.), followed by the raw JSON. That's just an example though, balance < expected balance is not appropriate condition as mentioned in here. So if the condition was "There is less than 48 hours worth of gas in the sponsor wallet" for example that should be the message instead.

Actions Taken​

Ali fixed the bug and updated the behaviour as needed.

Slack escalation link.

Incident Reviewer(s)​

  • Arda
  • Warren
  • Ali
  • Burak
Last updated on by Warren Anderson